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Background / Context 

SPARK is an early grade literacy and family engagement program developed by Boys & 
Girls Clubs of Greater Milwaukee. The tutoring component of SPARK is loosely based on the 
Reading Recovery program. Reading Recovery focuses on in-school tutoring with lesson plans 
written, and assessments, analyzed by the licensed teachers themselves, and has been found to be 
effective in developing student literacy skills (Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994). 
The family engagement aspect of SPARK is based on research that involving families in tutoring 
programs can improve children’s academic knowledge, skills and confidence (Bryan, 2005; 

Little, 2009). Encouraging family involvement in educational programs traditionally focuses on 
families attending events, receiving information from staff, volunteering (Epstein, 2001), and 
generally exhibiting “good parent” behaviors (Li, 2010). Getting to know families and the ways 
that their lives are structured outside of the educational setting may lead to a reciprocal 
relationship that can increase involvement (Graue & Hawkins, 2010). The family component of a 
program is not only to make families aware of the program’s mission and goals but also to 
empower families in their children’s learning both in the program and at home. 

Purpose / Objective / Research Question / Focus of Study 

The puipose of this report is to present the results of a two-year randomized control trial 
evaluation of the SPARK literacy program. SPARK is an early grade literacy program developed 
by Boys & Girls Clubs of Greater Milwaukee. In 2010, SPARK was awarded an Investing in 
Innovation (i3) Department of Education grant to further develop the program and test its impact 
in seven Milwaukee Public Schools (MPS). The evaluation tested the impact of SPARK across 
three domains: reading achievement, literacy, and school attendance. 

Setting 

The Milwaukee Public Schools (MPS), a district serving over 80,000 students, faces a 
significant challenge to teach its students how to read and write. Only 15% of MPS students 
were proficient in reading in the 201 1 administration of the Wisconsin Knowledge and Concepts 
Examination, compared to 35% statewide. The results of the WKCE are consistent with results 
of the National Assessment of Educational Progress (NAEP) and the ACT, which show that 
MPS students struggle with literacy throughout their education; only 15% of 4th grade MPS 
students are proficient in reading (NAEP, 2011) and 14% of MPS 11th graders scored at least 21 
on the ACT Reading Test, the benchmark identified for college readiness. The results of the 
NAEP further shows that there are significant achievement gaps for minority and low-income 
students. Among fourth grade students in MPS, 39% of white students are proficient in reading, 
compared to 7% of black students and 15% of Hispanic students. Only 7% of 4th grade low- 
income (free/reduced lunch participants) MPS students are proficient in reading, compared to 
48% of non-low-income students. 

These statistics demonstrate that the need for increased literacy opportunities in the 
Milwaukee area is urgent, and that this need is even more pronounced for low-income and 
minority students. SPARK was created in 2005 by Boys & Girls Clubs of Greater Milwaukee 
(BGCGM) to address this need. In 2010, SPARK received a Department of Education Investing 
in Innovation (i3) grant award to further develop the program and expand it to seven additional 
low-income and minority Milwaukee elementary schools. 

Participants 

In the fall of 2013, a total of 576 students across seven schools consented to participate in 
the SPARK program and evaluation. Selections were made in October and November, after fall 
assessments had been completed. 286 students were randomly selected as SPARK participants 
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and 290 as control students. 

Of the 576 consented students, 205 (36%) were in kindergarten, 214 (37%) in first grade, 
and 157 (27%) in second grade; 549 (95%) were eligible for free/reduced lunch; 459 (80%) were 
African American and 71 (12%) were Hispanic; 291 (50.5%) were female; and 51 (9%) had an 
IEP for speech or language disability. English language learners and students with learning 
disabilities, cognitive impairments, or emotional disabilities were excluded from the study but 
were able to receive SPARK tutoring. Table 1 presents the baseline literacy (PALS) and 
achievement scores (MAP) for the total consented sample of 576 students. 

Table 2 presents the samples used in each analysis and the corresponding attrition rates. 
Depending on the analysis, overall attrition rates ranged from 33.9% to 36.3%. These include 
187 students who moved away during the two years of the study and additional students with 
missing data. Differential attrition rates between participant and control students were not 
substantively different, ranging from 1.96% to 2.31%. There was no replacement of students 
who dropped from the evaluation. 

By the spring of the 2014-2015 school year, after attrition, the characteristics of the 
resulting sample of 389 students are presented in Table 3. The final sample consisted primarily 
of low-income and minority students. Table 4 presents the baseline PALS and MAP scores for 
the final sample. Table 5 presents final scores on the PALS, MAP, and school absences. 

Intervention / Program / Practice 

By using both in-school tutoring and family engagement, SPARK works both to develop 
the literacy skills of early-grade students and to support families as they learn to support the 
literacy development of their students. SPARK students are also encouraged to participate in 
after-school club activities. It is through this multi-modal strategy that SPARK seeks to have a 
lasting impact on students and to prepare them to succeed beyond their participation in SPARK. 
In-school tutoring 

The tutoring component of SPARK is loosely based on the Reading Recovery program. 
Reading Recovery, which focuses on in-school tutoring with lesson plans written by and 
assessments analyzed by the licensed teachers themselves, has been found to be effective in 
developing student literacy skills (Pinnell, Lyons, DeLord, Bryk, & Seltzer, 1994). Lor the in- 
school tutoring component of the current study, SPARK students were pulled out of non-core 
classes during the school day for 30 minutes, up to three times per week, for two years. In the 
current study, SPARK students received an intensive amount of services across the two program 
years while control group students received the “business as usual” reading instruction provided 
by MPS. The average SPARK student received 122.5 tutoring sessions (SD = 27.3). 

Family Engagement 

SPARK seeks to have a lasting impact on students by engaging families in the literacy 
development of students. To execute the family engagement component, each site has a parent 
partner who works with each participating student’s family. Their work is designed to bridge the 
divide between school and home by translating literacy concepts, educating families about a 
variety of literacy activities, and validating the literacy practices already happening in the home. 
Parent partners help families see how they already are incorporating literacy into their children’s 
lives and show parents how to promote literacy more effectively. Parent partners stay connected 
with families through a monthly newsletter, monthly family events at each site, phone calls, and 
emails. These communications are designed to keep families aware of student progress in 
SPARK, help families promote literacy at home, and address any school attendance issues that 
arise during the program. Parent partners also conduct home visits for all students twice during 
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the summer between their first and second year of participation and as needed during the school 
year. These visits are viewed as opportunities to connect with families in their own space and 
learn about the literacy activities already taking place in the home. Parents of SPARK students 
attended an average of three family events (Range 0 to 10, SD = 2.2), received 32 parent contacts 
(Range 0 to 69, SD = 13.9) and had 2.4 home visits (Range 0 to 8, SD = 1.4). 

Research Design 

The evaluation used a randomized control trial selection framework at the student level to 
identify the impact of SPARK. Informed consent was obtained from 576 parents for their 
students to participate in the study. A random sample of kindergarten, 1st, and 2nd grade 
students in seven MPS schools was selected in October and November of 2013 to participate. 

286 students were randomly selected as SPARK participants and 290 were randomly selected as 
control students. Stratification was done by school and grade level within school. The specific 
number of students selected to receive SPARK within each strata was determined both by the 
number of consented students and the capacity to serve students within each site. Students with a 
reading-related IEP or who were English Language Learners were not eligible to participate in 
the evaluation but were eligible to receive tutoring. All other students were eligible to participate. 

Data Collection and Analysis 

Outcome Measures 

The evaluation tested the impact of SPARK across three domains: reading achievement, 
literacy, and school attendance. All three outcomes listed below are collected by MPS and were 
provided directly to the evaluation by MPS for the purpose of evaluating SPARK. 

Reading achievement [Measures of Academic Progress (MAP) Reading Assessment! - The 
MAP is a norm-referenced, adaptive assessment of reading achievement. 

Literacy [Phonological Awareness Literacy Screening (PALS)l - The PALS is a criterion- 
referenced, teacher-administered assessment of literacy. MPS began administering the PALS to 
second grade students in the 2014-2015 school year. Administering the PALS to third grade 
students was optional, but MPS decided to do so in all SPARK schools so that the PALS could 
be used as an outcome in the evaluation for all grade levels. 

Regular School Data Attendance - The total number of absences for both the 2013-14 and 2014- 
2015 school years. 

Modeling Strategy 

Separate generalized linear statistical models with robust standard error estimators were 
used to compare spring 2015 MAP, spring 2015 PALS, and attendance (number of absences for 
both program years combined) of participants and controls for current first, second, and third 
grade students. The results of these grade level models were then pooled to estimate the overall 
impact of SPARK on each domain. 

Table 6 presents the covariates used in each of the grade-specific models for each 
outcome. Covariates were chosen based on their availability and predictive validity for each 
outcome. Baseline PALS scores were not available for 3rd grade students because MPS did not 
require schools to administer PALS to 2nd grade students. Because of this unavailability, MAP 
math scores were included as an additional covariate in 3rd grade MAP and PALS models. Non- 
predictive covariates were removed from each model. School was included as a fixed factor in all 
models to account for the clustering of data within schools. Other student characteristics such as 
ethnicity, gender, and IEP status were not found to uniquely predict outcomes and were not 
included in any models. Students with missing data were excluded from each analysis. All 
models were estimated using both standardized and unstandardized outcomes. The potential that 
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SPARK has a differential impact on students with different levels of baseline literacy and 
reading achievement was explored by including terms that interacted SPARK with baseline 
PALS (for 1st and 2nd grade models) and with baseline MAP reading (for 3rd grade models) in 
each model. When the interaction terms were found to be significant, additional models were 
estimated that split the sample in half by baseline literacy and achievement scores. 

Findings / Results 

SPARK was found to significantly impact student reading achievement. The overall, 
unstandardized effect of SPARK on the MAP was 2.8 RIT score points (Table 7). This 
corresponds to an effect size (Hedges g ) of .23. The pooled interaction term of SPARK 
participation and baseline PALS scores/MAP scores was also significant ( t = 2.17, p < .05) 
suggesting that SPARK differentially impacted MAP scores for students with different starting 
literacy and achievement levels. SPARK had a much larger impact on the reading achievement 
of lower-achieving students. The unstandardized effect of SPARK on these students was 4.4 RIT 
score points on the MAP (Table 7). This corresponds to an effect size (Hedge’s g ) of .36. 

SPARK was found to also significantly impact student literacy. The overall, 
unstandardized effect of SPARK on PALS was 5.7 scale points (Table 8). This corresponds to an 
effect size (Hedge’s g) of .35. The pooled interaction term of SPARK participation and baseline 
PALS scores/ MAP scores was again significant (t = 5.61, p < .001). The results of additional 
modeling suggests that SPARK again had a much larger impact on the literacy of less-literate 
students. The unstandardized effect of SPARK on PALS for these students in the lower half of 
achievement at baseline was 10.5 points (Table 8). This corresponds to an effect size (Hedge’s g) 
of .66. As an additional analysis of the PALS, scores were converted into literacy benchmarks, 
indicating whether students were on track in their literacy development. Table 9 presents the 
cross tabulation of baseline and post-SPARK benchmark status for first and second grade control 
and SPARK students. This table shows that there were 41 (21 SPARK and 20 control) students 
that started the evaluation below benchmark, while at post, there were 87 (35 SPARK and 52 
control). Nearly all (18) of the 20 control students that started below benchmark finished below 
benchmark, while most SPARK students, 13 out of 21, that started below benchmark finished at 
or above benchmark. 

Finally, SPARK was found to also significantly impact school attendance. The overall, 
unstandardized, effect of SPARK was 5.8 absences (Table 10), indicating that SPARK students 
had 5.8 fewer absences than control students. This corresponds to an effect size (Hedge’s g) of 
.25. The pooled interaction term of SPARK participation and baseline PALS scores/ MAP scores 
was not a significant predictor of school absences ( t = 1.04, p > .05). 

Conclusions 

The results suggest that SPARK had statistically significant, positive impacts on reading 
achievement, literacy, and regular school day attendance. SPARK students were absent from 
school, on average, 5.8 fewer times than control students. The benefit of SPARK was greater 
with students who started the program with greater literacy instruction needs; while only 10% of 
control students below literacy benchmarks at the start of the study met benchmark at the 
conclusion of the study, 62% of SPARK participants who started below benchmark met 
benchmark at the end of the study. 

Further research of SPARK should explore the effectiveness of SPARK with students 
outside of the Milwaukee context. The results here, although positive, only reflect one study. It is 
important the SPARK be tested in a variety of settings and contexts. Further, it will be important 
to follow SPARK students beyond their participation to determine the stability of the impact. 
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Appendix B. Tables and Figures 


Table 1: Fall 2013 MAP and PALS scores - Total sample 




MAP Reading RIT 

MAP Math RIT 

PALS 



Grade 

Mean 

SD 

n 

Mean 

SD 

n 

Mean 

SD 

n 

Control 

K 

142.9 

9.1 

104 

138.2 

9.8 

92 

64.0 

21.5 

93 


1st 

155.1 

10.6 

108 

155.8 

12.2 

94 

48.3 

19.3 

95 


2nd 

163.4 

12.4 

78 

167.7 

10.3 

75 





Total 

153.0 

13.4 

290 

153.0 

16.1 

261 

56.1 

21.8 

188 

SPARK 

K 

140.4 

10.6 

101 

137.4 

9.2 

89 

60.1 

22.6 

89 


1st 

155.3 

10.9 

106 

157.0 

12.5 

101 

50.0 

17.3 

101 


2nd 

165.8 

13.4 

79 

170.1 

10.7 

76 





Total 

152.9 

15.4 

286 

154.2 

17.0 

266 

54.7 

20.5 

190 

Total 

K 

141.6 

9.9 

205 

137.8 

9.5 

181 

62.1 

22.1 

182 


1st 

155.2 

10.7 

214 

156.4 

12.3 

195 

49.2 

18.3 

196 


2nd 

164.6 

12.9 

157 

168.9 

10.5 

151 





Total 

153.0 

14.4 

576 

153.6 

16.6 

527 

55.4 

21.2 

378 
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Table 2: Attrition rates for each analysis of the impact of SPARK 




Moved 

away 

Missing 

data 

Final 

sample 

Attrition 

rate 

MAP 

Control 

290 

96 

6 

188 

35.2% 


SPARK 

286 

91 

3 

192 

32.9% 


Total 

576 

187 

9 

380 

34.0% 

PALS 

Control 

290 

96 

12 

182 

37.2% 


SPARK 

286 

91 

10 

185 

35.3% 


Total 

576 

187 

22 

367 

36.3% 

Attendance 

Control 

290 

96 

5 

189 

34.8% 


SPARK 

286 

91 

3 

192 

32.9% 


Total 

576 

187 

8 

381 

33.9% 
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Table 3: Final sample of study participants 




Control 

SPARK 

Total 

Grade Level 

K-l st 

72 

66 

138 


l st -2 nd 

63 

74 

137 


2nd 

59 

55 

114 

School 

Brown 

27 

34 

61 


Cass 

21 

25 

46 


Clarke 

25 

21 

46 


81st 

27 

24 

51 


Engleburg 

32 

34 

66 


Rogers Street 

31 

32 

63 


Sherman 

31 

25 

56 

Race/Ethnicity 

Black 

149 

146 

295 


Hispanic 

29 

34 

63 


Other 

16 

15 

31 

Gender 

Female 

96 

100 

196 


Male 

98 

95 

193 

F/R Lunch Eligible 

No 

9 

8 

17 


Yes 

185 

187 

372 

IEP 

No 

182 

184 

366 


Yes 

12 

11 

23 

Total 


194 

195 

389 
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Table 4: Fall 2013 MAP and PALS scores - Final sample 




MAP Reading RIT 

MAP Math RIT 

PALS 



Grade 

Mean 

SD 

n 

Mean 

SD 

n 

Mean 

SD 

n 

Control 

K 

143.9 

9.0 

72 

139.0 

9.0 

69 

66.6 

20.7 

70 


1st 

156.2 

9.9 

63 

156.0 

11.5 

59 

51.3 

18.3 

60 


2nd 

163.1 

11.3 

59 

167.7 

9.6 

58 





Total 

153.7 

12.8 

194 

153.4 

15.6 

186 

59.5 

21.0 

130 

SPARK 

K 

141.4 

11.3 

66 

137.9 

9.7 

64 

59.7 

22.3 

64 


1st 

156.0 

11.7 

74 

156.8 

13.0 

74 

50.7 

17.4 

74 


2nd 

164.4 

12.5 

55 

168.5 

10.2 

54 





Total 

153.4 

15.0 

195 

153.8 

16.5 

192 

54.9 

20.2 

138 

Total 

K 

142.7 

10.2 

138 

138.5 

9.4 

133 

63.3 

21.6 

134 


1st 

156.1 

10.9 

137 

156.5 

12.3 

133 

51.0 

17.8 

134 


2nd 

163.7 

11.9 

114 

168.1 

9.8 

112 





Total 

153.6 

13.9 

389 

153.6 

16.1 

378 

57.2 

20.7 

268 
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Table 5: Spring 2015 MAP, PALS, and school absences - Final sample 


MAP Reading RIT PALS Absences 



Grade 

Mean 

SD 

n 

Mean 

SD 

n 

Mean 

SD 

n 

Control 

K 

173.7 

13.2 

71 

39.0 

17.8 

71 

32.8 

28.5 

71 


1st 

176.5 

15.3 

61 

51.8 

20.0 

59 

27.4 

24.4 

61 


2nd 

188.7 

12.5 

59 

66.6 

15.5 

55 

25.0 

21.1 

59 


Total 

179.2 

15.1 

191 

51.3 

21.1 

185 

28.7 

25.2 

191 

SPARK 

K 

173.0 

8.9 

65 

40.5 

13.3 

64 

29.8 

26.5 

65 


1st 

181.1 

15.2 

74 

61.1 

13.7 

70 

22.8 

17.1 

74 


2nd 

189.8 

14.8 

54 

68.0 

16.9 

53 

20.7 

14.2 

54 


Total 

180.8 

14.8 

193 

56.0 

18.5 

187 

24.6 

20.4 

193 

Total 

K 

173.3 

11.3 

136 

39.7 

15.7 

135 

31.4 

27.5 

136 


1st 

179.0 

15.4 

135 

56.8 

17.4 

129 

24.9 

20.7 

135 


2nd 

189.2 

13.6 

113 

67.3 

16.1 

108 

23.0 

18.2 

113 


Total 

180.0 

14.9 

384 

53.7 

19.9 

372 

26.6 

22.9 

384 
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Table 6: Factors included in each outcome model 


Outcomes 

2-year 

cohort 

SPARK 

School 

Fall 

2013 

PALS 

Fall 

2013 

MAP 

Reading 

Fall 

2013 

MAP 

Math 

F/R 

lunch 

School 

X 

PALS 

School 

xMAP 

Reading 

School 

xMAP 

Math 

PALS 

K- 1st 

X 

X 

X 

X 



X 

X 



1 st - 2nd 

X 

X 

X 

X 



X 

X 



2nd - 3rd* 

X 

X 


X 

X 



X 

X 

MAP 

K- 1st 

X 

X 

X 

X 



X 

X 



1 st - 2nd 

X 

X 

X 

X 



X 

X 



2nd - 3rd* 

X 

X 


X 

X 



X 

X 

Attendance 

K- 1st 

X 

X 

X 



X 

X 




1st - 2nd 

X 

X 

X 



X 

X 




2nd - 3rd* 

X 

X 


X 


X 


X 



* MPS did not administer the PALS with 2 nd grade students in the 2013-2014 school year. 
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Table 7: Effects of SPARK on MAP Reading scores 

B 


SE t-stat p-value 


Unstandardized 

Overall 

K 

1.99 

1.35 

1.47 




Lirst Grade 

6.04 

1.81 

3.33 




2nd Grade 

1.23 

1.70 

0.73 




Overall 

2.80 

0.67 

4.15 

<.001 


Low 

K 

2.56 

2.21 

1.15 




Lirst Grade 

9.00 

2.88 

3.12 




2nd Grade 

2.65 

3.13 

0.84 




Overall 

4.39 

1.11 

3.97 

<.001 


High 

K 

1.21 

1.68 

0.72 




Lirst Grade 

6.59 

2.53 

2.60 




2nd Grade 

1.94 

2.67 

0.73 




Overall 

2.66 

0.84 

3.18 

<.01 

Standardized 

Overall 

K 

0.18 

0.12 

1.47 




Lirst Grade 

0.39 

0.12 

3.33 




2nd Grade 

0.09 

0.12 

0.73 




Overall 

0.23 

0.06 

3.97 

<.001 


Low 

K 

0.23 

0.20 

1.16 




Lirst Grade 

0.59 

0.19 

3.12 




2nd Grade 

0.20 

0.23 

0.85 




Overall 

0.36 

0.09 

3.82 

<.001 


High 

K 

0.11 

0.15 

0.72 




Lirst Grade 

0.43 

0.16 

2.60 




2nd Grade 

0.14 

0.20 

0.73 




Overall 

0.23 

0.07 

3.07 

<.01 
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Table 8: Effects of SPARK on PALS scores 





B 

SE 

t-stat 

79 -value 

Unstandardized 

Overall 

K 

5.37 

2.00 

2.69 




Lirst Grade 

9.62 

2.47 

3.89 




2nd Grade 

2.53 

2.37 

1.07 




Overall 

5.69 

0.99 

5.76 

<.001 


Low 

K 

7.49 

2.91 

2.57 




Lirst Grade 

19.83 

3.76 

5.28 




2nd Grade 

2.18 

5.15 

0.42 




Overall 

10.47 

1.46 

7.19 

< .001 


High 

K 

-0.79 

2.23 

-0.35 




Lirst Grade 

1.05 

2.26 

0.46 




2nd Grade 

3.74 

1.36 

2.75 




Overall 

2.21 

0.92 

2.41 

< .01 

Standardized 

Overall 

K 

0.34 

0.13 

2.69 




Lirst Grade 

0.55 

0.14 

3.89 




2nd Grade 

0.16 

0.15 

1.07 




Overall 

0.35 

0.06 

5.69 

<.001 


Low 

K 

0.48 

0.19 

2.57 




Lirst Grade 

1.14 

0.22 

5.28 




2nd Grade 

0.14 

0.32 

0.42 




Overall 

0.66 

0.09 

7.09 

< .001 


High 

K 

-0.05 

0.14 

-0.35 




Lirst Grade 

0.06 

0.13 

0.46 




2nd Grade 

0.23 

0.08 

2.74 




Overall 

0.13 

0.06 

2.37 

< .05 
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Table 9: Cross-tabulation of students meeting PALS benchmark before and after SPARK 


Spring 2015 (Post SPARK) 




Below 

Met 

TV'vf ol 




benchmark 

benchmark 

I Oull 

Fall 2013 

Control 

Below benchmark 

18 

2 

20 

(Pre SPARK) 


Met benchmark 

34 

74 

108 


SPARK 

Below benchmark 

8 

13 

21 



Met benchmark 

27 

85 

112 


Total 

Below benchmark 

26 

15 

41 



Met benchmark 

61 

159 

220 



Total 

87 

174 

261 
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Table 10: Effects of SPARK on Attendance (Number of absences) 




B 

SE 

t-stat 

/;-value 

Unstandardized 

K 

5.6 

4.3 

1.31 



First Grade 

5.3 

3.7 

1.45 



2nd Grade 

6.6 

4.0 

1.63 



Overall 

5.8 

1.9 

3.00 

<.01 

Standardized 

K 

0.20 

0.15 

1.31 



First Grade 

0.26 

0.18 

1.45 



2nd Grade 

0.36 

0.22 

1.62 



Overall 

0.25 

0.08 

3.31 

<.001 


SREE Spring 2016 Conference Abstract Template 


B-10 



